LeRobot Datasets and Evaluation

LW-BenchHub collects and generates large-scale datasets via teleoperation that comply with the LeRobot specification, enabling out-of-the-box training and evaluation workflows. With the unified interface provided by EnvHub, developers can quickly build end-to-end experimental pipelines.

Dataset Overview

The Lightwheel Tasks LeRobot Datasets are divided into four subsets based on robot type, including ARX-X7s, Unitree G1, Agilex-Piper. The datasets following the LeRobot v3.0 common structure are available on HuggingFace Hub:

Dataset	Description	Tasks	Frames
Lightwheel-Tasks-X7S	X7S LIBERO and RoboCasa	117	~10.3M
Lightwheel-Tasks-Double-Piper	Double-Piper LIBERO	130	~6.0M
Lightwheel-Tasks-G1-Controller	G1-Controller LIBERO	62	~2.7M
Lightwheel-Tasks-G1-WBC	G1-WBC RoboCasa	32	~1.5M

For data visualization, you can view the dataset online through the official LeRobot Dataset Visualizer by entering the corresponding dataset repository ID. For example, the dataset repo ID for x7s robot is LightwheelAI/Lightwheel-Tasks-X7S

Evaluating Policies

Install LeRobot

LeRobot is an open-source framework for multimodal robot imitation learning and reinforcement learning, supporting training and evaluation of various policy models. You can set it up as follows:

git clone https://github.com/huggingface/lerobot.git
cd lerobot
pip install -e .
cd ..

# Revert numpy to version 1.26
pip install numpy==1.26.0 

Pre-trained Policies

The following trained policies are available:

Policy	Architecture	Task	Layout	Robot	Link
smolvla-double-piper-pnp	SmolVLA	L90K1PutTheBlackBowlOnThePlate	libero-1-1	DoublePiper-Abs	HuggingFace

Evaluate SmolVLA

Run lerobot_eval.sh to evaluate the pre-trained SmolVLA policy:

# cd lw_benchhub
lerobot-eval \
  --policy.path=LightwheelAI/smolvla-double-piper-pnp \
  --env.type=isaaclab_arena \
  --rename_map='{"observation.images.left_hand_camera_rgb": "observation.images.left_hand", "observation.images.right_hand_camera_rgb": "observation.images.right_hand", "observation.images.first_person_camera_rgb": "observation.images.first_person"}' \
  --env.hub_path=LightwheelAI/lw_benchhub_env \
  --env.kwargs='{"config_path": "configs/envhub/example.yml"}' \
  --trust_remote_code=true \
  --env.state_keys=joint_pos \
  --env.action_dim=12 \
  --env.camera_keys=left_hand_camera_rgb,right_hand_camera_rgb,first_person_camera_rgb \
  --policy.device=cuda \
  --eval.batch_size=10 \
  --eval.n_episodes=100

Environment Configuration

Evaluation can be quickly launched by modifying the robot, task, and layout settings in the configuration file.

Full Configuration Options

# =========================
# Basic Settings
# =========================
disable_fabric: false
device: cuda:0
sensitivity: 1.0
step_hz: 50
enable_cameras: true
execute_mode: eval
episode_length_s: 20.0 # Episode length in seconds, increase if episodes timeout during eval

# =========================
# Robot Settings
# =========================
robot: DoublePiper-Abs # Robot type, DoublePiper-Abs, X7S-Abs, G1-Controller or G1-Controller-DecoupledWBC
robot_scale: 1.0

# =========================
# Task & Scene Settings
# =========================
task: L90K1PutTheBlackBowlOnThePlate # Task name
scene_backend: robocasa
task_backend: robocasa
debug_assets: null
layout: libero-1-1 # Layout and style ID
sources:
  - objaverse
  - lightwheel
  - aigen_objs
object_projects: []
usd_simplify: false
seed: 42

# =========================
# Object Placement Retry Settings
# =========================
max_scene_retry: 4
max_object_placement_retry: 3

resample_objects_placement_on_reset: true
resample_robot_placement_on_reset: true

# =========================
# Replay Configuration Settings
# =========================
replay_cfgs:
  add_camera_to_observation: true
  render_resolution: [640, 480]

Note

• Task and Layout Configuration:

  • The task should be changed to the corresponding task name.
  • For Lightwheel-LIBERO-Tasks: set the layout to the corresponding layout number, such as libero-1-1, since LIBERO tasks use fixed scenes.
  • For Lightwheel-RoboCasa-Tasks: the numeric layout suffix can be omitted to enable randomized layouts.

• Image Resolution:

  • replay_cfgs.render_resolution determines the resolution (width × height) of the returned images.
  • This should match the input size used during training.

See Also

• LightwheelAI HuggingFace Datasets